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section addresses the following questions related to usage studies of 
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libraries conduct transaction log analysis? (3) How do libraries conduct 
transaction log analysis? (4) Who uses the results of transaction log 
analysis? and (5) What are the issues, problems, and challenges with 
transaction log analysis? The fourth section covers general issues and 
challenges in planning and implementing a research project. The fifth section 
presents conclusions and considers future directions. Appendices include: 
references and a selected bibliography; a list of participating institutions; 
the survey questions; and traditional input, output, and outcome measures. 
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Preface 

Making library services available online is not only expensive; it is also very 
risky. The library’s roles there are not at all clear. Neither are its relationships 
with users or with other information services. There is little information 
about how library users behave in a network environment, how they react to 
online library services, and how they combine those services with others such 
as search engines like Google, bookstores like Amazon, Internet gateways like 
Voice of the Shuttle, and instructional technologies like WebCT or Blackboard. 
Digital libraries are still relatively immature— most are still at a stage where 
limited experimentation is more important than well-informed strategic plan- 
ning. While libraries have excelled at assessing the development and use of 
their traditional collections and services, comparable assessments of online 
collections and services are more complicated and less well understood. 

Against this backdrop, the Digital Library Federation (DLF) has commit- 
ted to driving forward a research process that will provide the information 
that libraries need to inform their development in a networked era. The goals 
of this process are: 

• to develop a better understanding of methods effective in assessing use 
and usability of online scholarly information resources and information 
services; and 

• to create a baseline understanding of users’ needs to support strategic 
planning in an increasingly competitive environment for academic librar- 
ies and their parent institutions. 

This report is an initial step in achieving the first of these goals. It offers a 
survey of the methods that are being deployed at leading digital libraries to 
assess the use and usability of their online collections and services. Focusing 
on 24 DLF member libraries, the study’s author, Distinguished DLF Fellow 
Denise Troll Covey, conducted numerous interviews with library profession- 
als who are engaged in assessment. In these interviews, Covey sought to doc- 
ument the following: 

• why digital libraries assessed the use and usability of their online collec- 
tions and services 

• what aspects of those collections and services they were most interested in 
assessing 

• what methods the libraries used to conduct their assessments 

• which methods worked well and which worked poorly in particular kinds 
of assessments 

• how assessment data were used by the library, and to what end 

• what challenges libraries faced in conducting effective assessments 
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The result is a report on the application, strengths, and weaknesses of as- 
sessment techniques that include surveys, focus groups, user protocols, and 
transaction log analysis. Covey s work is also an essential methodological 
guidebook. For each method that she covers, she is careful to supply a defini- 
tion, explain why and how libraries use the method, what they do with the 
results, and what problems they encounter. The report includes an extensive 
bibliography on more detailed methodological information, and descriptions 
of assessment instruments that have proved particularly effective. Examples 
are available on the Web for all to see, and potentially to modify and use. The 
work concludes with a review of the challenges that libraries face as they seek 
to gather and use reliable information about how their online presence is felt. 
These concluding remarks will be of general interest and are recommended to 
senior library managers as well as to those more directly involved with as- 
sessment activities. 

Given its practical orientation, Usage and Usability is an ideal launching 
pad for CLIR’s new series, Tools for Practitioners. The series emphasizes the 
immediate, the practical, and the methodological. As it develops, it will in- 
clude work that, like Coveys, appeals to and provides guidance for particu- 
lar professional audiences. 



Daniel Greenstein 

Director ; Digital Library Federation 
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1. INTRODUCTION 



A s the needs and expectations of library users change in the 
digital environment, libraries are trying to find the best 
ways to define their user communities, understand what 
they value, and evolve digital library collections and services to meet 
their demands. In part, this effort requires a closer, more formal look 
at how library patrons use and respond to online collections and 
services. 

To synthesize and learn from the experiences of leading digital 
libraries in assessing use and usability of online collections and ser- 
vices, the Digital Library Federation (DLF) undertook a survey of its 
members. From November 2000 through February 2001, the author 
conducted interviews with 71 individuals at 24 of the 26 DLF mem- 
ber institutions (representing an 86 percent response rate at the 24 
institutions). Participants were asked a standard set of open-ended 
questions about the kinds of assessments they were conducting; 
what they did with the results; and what worked well or not so well. 
Follow-up questions varied, based on the work being done at the in- 
stitution; in effect, the interviews tracked the efforts and experiences 
of those being interviewed. 

The results of the survey reveal the assessment practices and 
concerns of leading digital libraries. They are not representative of all 
library efforts; however, they do show trends that are likely to inform 
library practice. The study offers a qualitative, rather than quantita- 
tive, assessment of issues and practices in usage and usability data 
gathering, analysis, interpretation, and application. 



1.1. Report Structure 

The survey indicates significant challenges to assessing use and us- 
ability of digital collections and services. The rest of Section 1 sum- 
marizes these challenges. Subsequent sections elaborate on these 
challenges and draw on examples from the assessment efforts of DLF 
libraries. Sections 2 and 3 describe libraries’ experiences using popu- 
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lar methods to conduct user studies, such as surveys, focus groups, 
user protocols, and transaction log analysis. The report explains 
what each of these methods entails, its advantages and disadvantag- 
es, why and how libraries use it, the problems encountered, and the 
lessons libraries have learned from experience. Section 4 covers gen- 
eral issues and challenges in conducting research, including sam- 
pling and recruiting representative research subjects, getting Institu- 
tional Review Board (IRB) approval to conduct research with human 
subjects, and preserving user privacy. Section 5 summarizes the con- 
clusions of the study and suggests an agenda for future discussion 
and research. Appendix A provides a selected bibliography. A list of 
institutions participating in the survey appears in Appendix B, while 
Appendix C lists the interview questions. An overview of more tra- 
ditional library input, output, and outcome assessment efforts, and 
the impact of digital libraries on these efforts, is provided in Appen- 
dix D; this information is designed to help the reader position the 
information in this report within the context of library assessment 
practices more generally. 

To preserve the anonymity of DLF survey respondents and re- 
spect the sensitivity of the research findings, the report does not as- 
sociate institution names with particular research projects, incidents, 
or results. The word “faculty” is used to refer to teachers and profes- 
sors of for-credit academic courses. The word “librarian” is used, re- 
gardless of whether librarians have faculty status in their institutions, 
or, indeed, whether they hold an MLS degree. 



1.2. Summary of Challenges in Assessment 

DLF respondents shared the following concerns about the efficiency 
and efficacy of their assessment efforts: 

• Focusing efforts to collect only meaningful, purposeful data 

• Developing the skills to gather, analyze, interpret, present, and 
use data 

• Developing comprehensive assessment plans 

• Organizing assessment as a core activity 

• Compiling and managing assessment data 

• Acquiring sufficient information about the environment to under- 
stand trends in library use 

Collecting only meaningful, purposeful data. Libraries are 
struggling to find the right measures on which to base their deci- 
sions. DLF respondents expressed concern that data are being gath- 
ered for historical reasons or because they are easy to gather, rather 
than because they serve useful, articulated purposes. They ques- 
tioned whether the sheer volume of data being gathered prohibits 
their careful analysis and whether data are being used to their full 
advantage. Working with data is essential, time-consuming, and 
costly-so costly that libraries are beginning to question, and in some 
cases even measure, the costs and benefits of gathering and analyz- 
ing different data. Respondents know that they need new measures 
and composite measures to capture the extent of their activities in 
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both the digital and traditional realms. Adding new measures is 
prompting many DLF sites to review their data-gathering practices. 
The libraries are considering, beginning, or completing needs assess- 
ments of the data they currently gather, or think they should gather, 
for internal and external purposes. If such information is not needed 
for national surveys or not useful for strategic purposes, chances are 
it will no longer be gathered, or at least not gathered routinely. How- 
ever, deciding what data should be gathered is fraught with difficul- 
ties. Trying to define and measure use of services and collections that 
are rapidly changing is a challenge. The fact that assessment meth- 
ods evolve at a much slower rate than do the activities or processes 
they are intended to assess compounds the problem. How can librar- 
ies measure what they do, how much they do, or how well they do, 
when the boundaries keep changing? 

Developing skills to gather, analyze, interpret, present, and use 
data. Several DLF respondents commented that they spend a great 
deal of time gathering data but do not have the time or talent to do 
anything with this information. Even if libraries gather the right 
measures for their purposes, developing the requisite skills to ana- 
lyze, interpret, present, and use the data are separate challenges. For 
example, how do you intelligibly present monthly usage reports on 
8,000 electronic journals? The answer is you don’t. Instead, you 
present the statistics on the top 10 journals, even though this severely 
limits the dissemination and application of data painstakingly gath- 
ered and compiled. Though DLF respondents indicated that they are 
learning slowly from experience how to make each research method 
work better for their purposes, many said they need methodological 
guidance. They need to know what sampling and research methods 
are available to recruit research subjects and assess use and usability 
of the digital library, which methods are best suited for which pur- 
poses, and how to analyze, interpret, present, and use the quantita- 
tive and qualitative data they gather to make effective decisions and 
strategic plans. 

Developing comprehensive assessment plans. Planning assess- 
ment from conception through follow-up also presents challenges. 
Ideally, the research process should flow seamlessly — from deciding 
to gather data to developing and implementing plans to use the data. 
In reality, however, DLF respondents reported frequent breakdowns 
in this process. Breakdowns occur for a number of reasons. It may be 
that something went awry in the planning or scheduling of the 
study. People assigned responsibility for certain steps in the process 
may lack the requisite skills. Staff turnover or competing priorities 
may intervene. Respondents also made it clear that the more people 
involved in the research process, the longer it takes. The longer the 
process takes, the more likely it is that the results will be out of date, 
momentum will be lost, or other phenomena will intrude before the 
results are implemented. Finally, if the study findings go unused, 
there will be less enthusiasm for the next study, and participation is 
likely to decrease. This applies both to the people conducting the 
study and to the research subjects. Conducting a study creates expec- 




10 



4 



Denise Troll Covey 



tations that something will be done with the results. When the re- 
sults are not applied, morale takes a hit and human and financial re- 
sources are wasted. Participants lose confidence, and the study plan- 
ners lose credibility. 

Organizing assessment as a core activity. DLF respondents well 
understood that in an environment of rapid change and limited re- 
sources, libraries cannot afford these outcomes from their assessment 
efforts. They also seemed to understand that the way in which an 
assessment is organized affects the outcome. At some institutions, 
user studies are centralized and performed by recently hired experts 
in the field. At others, user studies are decentralized and performed 
systemwide; they involve efforts to teach librarians and staff 
throughout the organization how to conduct research using different 
methods. Still other institutions, sparked by the interests of different 
personnel, take an ad hoc approach to user studies. A few libraries 
have established usability testing programs and laboratories. If the 
goal is a culture of assessment, then making assessment a core activi- 
ty and allocating human and financial resources to it is essential. The 
key is not how a study is organized, but that it is organized and sup- 
ported by commitment from administrators and librarians. Com- 
ments from DLF respondents suggested that given sufficient human 
and financial resources, requisite skills could be acquired, guidelines 
and best practices developed, and assessments conducted routinely, 
efficiently, and effectively enough to keep pace with the pace of 
change. 

Compiling and managing assessment data. Many DLF respon- 
dents expressed concern about the effort required to compile and 
manage data collected by different people and assessments. Libraries 
need a simple way to record and analyze quantitative and qualitative 
data and to generate statistical reports and trend lines. Several DLF 
sites have developed or are developing a management information 
system (MIS) to compile and manage statistical data. They are wres- 
tling with questions about how long data should be kept, how data 
should be archived, and whether one system can or should manage 
data from different kinds of assessments. Existing systems typically 
have a limited scope. For example, one site has a homegrown desk- 
top reporting tool that enables library staff to generate ad hoc reports 
from data extracted and to update them regularly from the integrat- 
ed library system. Users can query the data and run cross-tabula- 
tions. The tool is used for a variety of purposes, including analysis of 
collection development, materials expenditures, and the productivity 
of the cataloging department. Reports can be printed, saved, or im- 
ported into spreadsheets or other applications for further analysis or 
manipulation. New systems being developed appear to be more 
comprehensive; for example, they attempt to assemble statistical data 
from all library departments. The ability to conduct cross-tabulations 
of data from different departments and easily generate graphics and 
multiyear trend lines are important features of the new systems. 

Acquiring sufficient information about the environment to un- 
derstand trends in library use. Several DLF respondents noted that 
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emerging new measures will assess how library use is changing in 
the networked environment, but these measures will not explain why 
library use is changing. Academic libraries need to know how stu- 
dents and faculty find information, what resources they use that the 
libraries do not provide, why they use these resources, and what 
they do with the information after they find it. This knowledge 
would provide a context for interpreting existing data on shifting 
patterns of library use and facilitate the development of collections, 
services, and tools that better meet user needs and expectations. Li- 
brary user studies naturally focus on the use and usability of library 
collections, services, and Web sites. The larger environment remains 
unexplored. 



2. USER STUDIES 



DLF respondents devoted the bulk of their discussion to user stud- 
ies, reflecting the user-centered focus of their operations. One re- 
spondent referred to the results of user studies as “outcome” mea- 
sures because, although they do not measure the impact of library 
use on student learning or faculty research, they do indicate the im- 
pact of library services, collections, facilities, and staff on user experi- 
ences and perceptions. 

Libraries participating in the DLF survey organize, staff, and 
conduct user studies differently. Some take an ad hoc approach; oth- 
ers use a more systematic approach. Some sites have dedicated staff 
experts in research methodologies who conduct user studies; others 
train staff throughout the libraries to conduct user studies. Some li- 
braries take both approaches. Some have consulted experts on their 
campuses or contracted with commercial firms to develop research 
instruments and analyze the results. For example, libraries partici- 
pating in the DLF survey have recruited students in library science 
and human-computer interaction to conduct user studies or hired 
companies such as Websurveyor.com or Zoomerang.com to host 
Web-based surveys and analyze the data. Libraries that conduct user 
studies use spreadsheet, database, or statistical analysis software to 
manage and analyze the data. In the absence of standard instru- 
ments, guidelines, or best practices, institutions either adapt pub- 
lished efforts to local circumstances or make their own. There is 
clearly a flurry of activity, some of it not well organized or effective, 
for various reasons discussed elsewhere in this report. 

Learning how to prepare research instruments, analyze and in- 
terpret the data, and use the results is a slow process. Unfortunately, 
however, the ability to quickly apply research results is often essen- 
tial, because the environment changes quickly and results go out of 
date. Many DLF respondents reported instances where data lan- 
guished without being analyzed or applied. They strongly cautioned 
against conducting research when resources and interest are insuffi- 
cient to support use of the results. Nevertheless, DLF libraries are 
conducting many user studies employing a variety of research meth- 
ods. The results of these studies run the gamut: they may reinforce 
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